Uncertainty in quantitative health impact modelling

Christopher Jackson
MRC Biostatistics Unit, University of Cambridge

Uncertainty in HIA models: Summary of lecture

  • Parametric models — definitions

  • Quantifying parameter uncertainty using probability distributions

  • Questions we can answer better by quantifying uncertainty

    • Uncertainty analysis — how confident are our results, how strong is the evidence?

    • Sensitivity analysis — how would results change if part of the model changed?

    • Value of Information (VoI) analysis — what parts of the model do we need better evidence for?

Quantitative models

Models approximate process that generates some output of interest, helping to inform decision-making.

Example: ITHIM health impact model [ TODO borrow picture ]

  • inputs: travel data, air pollution, physical activity, injuries, + scenarios of change

  • outputs: population health (and changes under scenarios)

These generally involve parameters. What are these?

Parametric models

Model input parameters: examples

  • background exposure [measures of average e.g. air pollution, physical activity] in an area

  • relative risk for [health outcome] given [change in AP / exposure]

Parameters are representations of knowledge about a (real or imagined) population.

  • Usually conceive as summaries of individual quantities over a large/infinite “population”.

Model input parameters \(\rightarrow\) model output quantities.
Model outputs are usually summaries over populations

Parameter uncertainty

Knowledge is often uncertain \(\rightarrow\)

  • parameters are uncertain

  • conclusions from models are uncertain

Individual variability versus uncertainty about knowledge

Example parameters, for some population

  • Background exposure to PM2.5.

  • Risk of death within one year.

We may be sure about the values of these. Even so…

  • Exact exposures will be different for each individual

  • Some individuals will die within a year, some won’t

But we may be uncertain about the parameter values.

Individual variability can never be removed, but parameter uncertainty can be reduced with better knowledge

Parameters are generally uncertain

Two broad reasons for parameter uncertainty

  • summary of a limited population

  • population is different from the one we want

- Discuss examples of this 

Models are also uncertain (“structural” uncertainty, “all models are wrong, but some are useful”…)

  • Ideally quantify as many uncertainties as possible inside the model

Why does uncertainty matter?

[ interact ? ]

We’ve built best model we can. Just report our “best estimate” to the decision maker?

  • Decision makers need good evidence to change practice — models should indicate strength of evidence for result

  • What about the future - we may be able to get better evidence - what research should be done?

Here we will cover quantitative methods for assessing uncertainty. In particular, probabilistic methods.

  • though there is a broader field of appraising evidence, engaging with stakeholders or experts etc…

Questions relevant to uncertainty quantification

Uncertainty analysis: about strength of evidence

  • What range of outputs are plausible, given current evidence?

  • Which parameters most influence the output uncertainty?

Sensitivity analysis: “what if…”

  • What if parameter took the value \(b\) instead of \(a\) — how would the output change?

Value of Information analysis

  • How much would the model improve if we got better information?
    Which parameter should we get better information on?

How to quantify uncertainty

Two broad approaches

  1. Statistical analyses of data (your own, or published analyses) giving point and interval estimates or standard errors

  2. Judgements (informal or from structured elicitation), e.g.

  • point estimate (best guess) for parameter value

  • credible interval e.g. “I judge that the parameter is between \(a\) and \(b\), with 95% confidence”

Our goal in each case is to obtain probability distributions for parameters

  • rigorous basis for uncertainty and sensitivity analysis

Probability distributions

A full probability distribution

For any pair of values \((a,b)\), we can deduce the probability that the parameter is between \(a\) and \(b\).

Statistical analyses: Bayesian methods

In statistical models, data assumed to come from models with parameters…

e.g. number of disease cases is Binomial (population n, underlying prevalence p)

  • Bayesian methods based around quantifying parameter uncertainties as probability distributions

  • Prior distribution combined with study data \(\rightarrow\) posterior distribution.

Prior distribution dominates if data are weak
Prior doesn’t matter if enough data

Statistical analyses: frequentist methods

  • Construct an estimator of parameters based on a finite dataset

  • Uncertainty quantified by imagining different datasets drawn from the same population

    • Standard error: variability in estimates between datasets.

    • Confidence interval: contains true value in 95% of draws.

If dataset is large, can interpret a frequentist analysis as Bayesian: parameter has a normal distribution, defined by estimate and standard error

If dataset is small, Bayesian analyses have the benefit of allowing background information to be included as a prior

Examples of statistical models informing QHIA

Global

disbayes informing incidence, CF

Meta analysis of relative risks

Local

(transport) travel time speeds convert to distances

Road traffic injury models - Poisson

Examples of judgements

Example: MMET values for transport-related cycling

Which of these are relevant to our particular health impact model — pick 6.8 (4, 10) given judgement?

Examples of judgements

Example: PM2.5 concentration. [SHOW APnA Table 3.] We have average and SD for 30 cities in India.

We have average for our city, but we want a measure of uncertainty

Could choose our SD as average of these SDs (?? geometric average (exp(mean(log))) or median?)

But what do we know about how our city compares to these?

Is it worth woing into effort to quantify — start with conservative assessment of uncertainty

Considerations: * size and quality of data underlying * how was it measured * relevance of population * changes over time * changes over space

Where might these come from with no extra info?

Meta-analysis

Formally averaging information from different studies.

Most developed in randomised clinical trials — highly controlled, regulated studies.

Give more weight to more confident studies and/or those closer to our context.

TODO any resources for MA of clinical trials and epidemiology. R package at least

PICTURE of meta-analysis

Uncertainty about a predicted new study – more appropriate than the average

Distributions from credible intervals

Suppose we have a parameter with estimate 0, credible interval (-2 to 2)

What is wrong with a distribution like this

Distributions from credible intervals

Suppose we have a parameter with estimate 0, credible interval (-2 to 2)

A triangular distribution is a bit more plausible

Distributions from credible intervals

Suppose we have a parameter with estimate 0, credible interval (-2 to 2)

A normal distribution is even better

Normal distribution

Used for quantities with unrestricted ranges

Defined by mean \(\mu\) and standard deviation \(\sigma\) (or variance \(\sigma^2\))

95% credible interval is \(\pm 2\) SDs: i.e. width is 4 SDs.: SD easily derived from a CI.

Log-normal distribution

Used for positive-valued quantities.

Normal distribution for the log of the quantity

Example: MMET/h estimate 2.5 (CI 1 to 4).

  • Transform to log(MMET).

  • Estimate log(2.5), CI (log(1) to log(4)).

  • Assign normal with SD = CI width / 4

If the SD is published, but not the CI, e.g. MMET/h estimate \(m=2.5\) (SD \(s=1.4\))?

  • Method of moments gets us mean \(\mu\) and SD \(\sigma\) on log scale \(\mu = \log(m/\sqrt{s^2/m^2 + 1}), \sigma= \sqrt{\log(s^2/m^2 + 1)}\)

Beta distribution

Used for quantities between 0 and 1: probabilities, proportions

Two “shape” parameters \(a,b\). Mean: \(a/(a+b)\).

Given an estimate \(m\) and credible interval, could approximate SD \(s =\) CI width/4 and use method of moments: \(a = (m(1-m)/s^2 - 1)m, b = (m(1-m)/s^2 - 1)(1 - m)\)

Example (a): estimate 0.4, 95% CI 0.2 to 0.6

Example (b): estimate 0.04, 95 %CI 0.01 to 0.4

Distribution (b) is a worse approximation to our desired CI width
“width/4” heuristic based on symmetric, normal-shaped distributions.
Could informally calibrate to match desired belief more closely.

See SHELF R package for more sophisticated techniques for fitting distributions

Practical - quantifying uncertainty around parameters

[ reference to ITHIM ]

Doing uncertainty analysis: Monte Carlo simulation

For each \(i = 1, 2, \ldots N\) (enough to give precise summaries)

  1. Simulate parameters \(X_i\) from their uncertainty distributions

  2. Compute the model output \(Y_i = g(X_i)\)

producing a sample from the model outputs \(Y_1, \ldots, Y_N\) [ animate? ]

Summarise the sample to give e.g. 

  • a credible interval for the outputs

  • probability that e.g. number of deaths \(>\) [important value]

Technical point - uncertainty affects “best estimate” in nonlinear models

Given a model with inputs \(X\) and outputs \(Y = g(X)\), where \(g()\) is some mathematical function.

Inputs \(X\) are uncertain, with expected values \(E(X)\).

What is the expected value of \(Y\)? Can we just plug in the best estimates of the inputs \(g(E(X)\)?

\(E(Y)\) is only equal to \(g(E(x))\) if the function \(g()\) is linear:

\[g(E(x)) = g((X_1 + \ldots + X_n)/n) = \] (if \(g()\) linear…)

\[(g(X_1) + \ldots + g(X_n)) / n = E(g(X)) = E(Y)\]

Most realistic models are **non-linear*: need Monte Carlo simulation to get the true expectation of the output.

One-way sensitivity analysis

Answers questions like “what if [parameter] was actually \(b\) instead of \(a\)

For each parameter, compare model output with

  • parameter at a “low value”

  • parameter at a “high value”

What if all the other parameters are uncertain? Probabilistic one-way sensitivity analysis:

  • Fix [parameter] at high or low value

  • Run the model under Monte Carlo analysis, using the uncertainty distributions of the other parameters

Tornado plot TODO nicer

Practical - uncertainty analysis in a model

[ reference to ITHIM ]

Value of Information

Recall

  • sensitivity analysis: “what if model was a bit different”

  • uncertainty analysis: “what is strength of evidence in model”

A different (related) question: “what would be the benefit of getting better information”.

This is Value of Information analysis

[Example - connect to a result from probabilistic 1 way SA. Looks like varying ? within plausible range has more of an effect than… so we might want to get better info ]

Value of Information

Given current information \(x\) model output is uncertain. But how much more precise would it get…

if we were to learn some parameter exactly?

  • Expected value of partial perfect information

if we conducted a study (say, a survey of 100 people to estimate it

  • Expected value of sample information

Helps us to:
set research priorities to reduce uncertainty
design studies (more advanced, not covered here)

Value of Information in terms of health economics

Value of Information methods developed in health economics

  • Model a health policy decision and its consequences (e.g. health benefits as QALYs vs costs). Parameter uncertainties as probability distributions

  • Information has value: \(\rightarrow\) reduces parameter uncertainty \(\rightarrow\) more precise model outputs \(\rightarrow\) better informed policy-making \(\rightarrow\) health benefits

Not previously used much in health impact modelling, where models used more for scenarios than policies

How can we define “value” if we don’t model a health policy?

Precision as value

Define value as “precision of estimate” (e.g. health impact of scenario)

  • How much will the variance (or SD, or credible interval width) be expected to reduce if we got better information?

  • More precise estimates (implicitly) assumed to ultimately lead to benefits

  • If needed, trade off informally with costs of research (willingness to pay for more precise estimates? Not covered here)

How exactly to do these computations?

Computing expected value of partial perfect information (EVPPI)

Model \(Y = g(X_1, X_2,.. )\) with some (scalar) output \(Y\) and multiple inputs \(X_r\)

Do uncertainty analysis: define distributions for each \(X_r\), obtain distribution for \(Y\) via Monte Carlo simulation.

\(var(Y)\): variance of model output under current information

Definition of EVPPI - expected reduction in this variance if we were to learn the exact value of \(X_r\) (say)

\[var(Y) - E_x(var(Y | X_r = x))\]

We don’t know the value of \(X_r\) when we calculate this — so we must take the expectation over possible values \(x\) (using the distribution we defined)

Illustration of EVPPI computation

Take the Monte Carlo sample, with all parameters uncertain

Regression of \(Y\) = model output versus \(X\) = parameter of interest

Expected reduction in variance of \(Y\) if we learnt \(X\)

\[var(Y) - E_x(var(Y | X = x))\]

We don’t know \(X\) so we average over our uncertainty

\(var(Y)\): variance of \(Y\)

\(E_x(var(Y | X = x))\): residual variance: mean of (“observed” - fitted) over different \(x\)

Spline regression for computing EVPPI

Regression function of output on input will not necessarily be linear.

Spline regression (automated choice of smoothness, works well enough in this context)

lm(Y ~ X)  # linear model
library(mgcv)
gam(Y ~ X) # spline ("generalized additive") model

The voi package can take care of extracting ingredients needed for the EVPPI computation (see the practivel)

library(voi)
evppivar(outputs, inputs)

EVPPI for more than one parameter at once

Example: dose-response curve governed by four different parameters \(\alpha\), \(\beta\), \(\gamma\), \(\tau\). Estimate the expected value of jointly learning all four of them (hence learning the dose-response curve)

Regression model with four predictors, e.g.

gam(Y ~ alpha + beta + gamma + tau) 

More advanced regression models available, which automatically choose the best fitting regression function for Y given the predictors (and their interactions): see the practical about the voi package

Presenting EVPPI results

Impact of different parameters on the output uncertainty, presented as

Uncertainty that would remain in the output \(Y\) if we knew \(X\)

  • as a variance, \(var(Y) - EVPPI_X\)

  • …or standard deviation (square root of this)

  • …or rough credible interval (\(\pm 2\) remaining standard deviation)

or proportion of uncertainty (variance) in \(Y\) explained by \(X\)

Value of Information: further topics

We will never get perfect information, but we may get better information via a new research study.

Expected value of sample information is the expected value of a study of a particular design / sample size

  • Slightly harder to define and compute

  • Previously used in health economic decision models, but not (yet!) health impact models

  • See voi package and references there for some examples.

Value of Information: further topics

Here we have just talked about “value” as variance of outputs

Value of Information also used widely for health economic decision models

  • Model chooses policy with the highest net benefit (or lowest loss)

  • “Value” is the net benefit (NB) itself

  • Value of information is NB(better information) - NB(current information)

See voi package and references there for some examples.

Practical session

… Use of “voi” package

Group discussion of advanced topics

What uncertainty/sensitivity questions do you want to answer in your own work?

Do you have the tools to do this?